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Abstract 

It is shown that an exact solution of the transient dynamics of an associative memory model 
storing an infinite number of limit cycles with I finite steps by means of the path-integral analysis. 
Assuming the Maxwell construction ansatz, we have succeeded in deriving the stationary state 
equations of the order parameters from the macroscopic recursive equations with respect to the 
finite-step sequence processing model which has retarded self-interactions. We have also derived 
the stationary state equations by means of the signal-to-noise analysis (SCSNA). The signal-to- 
noise analysis must assume that crosstalk noise of an input to spins obeys a Gaussian distribution. 
On the other hand, the path-integral method does not require such a Gaussian approximation of 
crosstalk noise. We have found that both the signal-to-noise analysis and the path-integral analysis 
give the completely same result with respect to the stationary state in the case where the dynamics 
is deterministic, when we assume the Maxwell construction ansatz. 

We have shown the dependence of storage capacity (oc) on the number of patterns per one limit 
cycle (/). At / = 1, storage capacity is Uc = 0.138 like the Hopfield model's. Storage capacity 
monotonously increases with the number of steps, and converges to Oc = 0.269 at I ~ 10. The 
original properties of the finite-step sequence processing model appear as long as the number of 
steps of the limit cycle has order / = 0(1). 



* Electronic address: |mimur a@ cs . hir oshima- cu. ac . j p| 
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I. INTRODUCTION 



In recent years, theories that can analyze the transient dynamics have been discussed for 
systems with frustrations especially a correlation-type associative memory 
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10|,|ll|. Diiring et al presented a path- integral method for an infinite-step sequence 
processing model and analyzed the properties of the stationary state [l^. By using During et 
al's analysis, Kawamura and Okada succeeded in deriving an exact macroscopic description 
of the transient dynamics Il3l . The transient dynamics can be analyzed not only by using 
the path- integral method but by using the signal-to-noise analysis, e.g., statistical 



neurodynamics 



161 Qd The signal-to -noise analysis is an approximation theory in which 



crosstalk noise obeys a Gaussian distribution. On the other hand, the path-integral method 
does not require such a Gaussian approximation of crosstalk noise. However surprisingly, 
the macroscopic equations of the exact solution given by means of the path-integral method 
are completely equivalent to those of the signal-to-noise analysis with respect to this model. 

It has turned out that the infinite-step sequence processing model can be more easily 
analyzed than the Hopfield model even if it is necessary to treat the dynamical process 
directly. This reason is as follows. The retrieval state of the infinite-step sequence processing 
model has no equilibrium state. Therefore, the correlations of the system are not very 
complex. Since the Hopfield model takes the same states repeatedly, its statistical properties 
are more complex than the infinite-step sequence processing model. Gardner et al analyzed 
the transient dynamics of the Hopfield model by using the path-integral method in the 
case where the dynamics is deterministic They obtained the macroscopic equations of 
the transient dynamics at time step t using O(t^) macroscopic variables and also obtained 
the macroscopic equations of the equilibrium state from the transient dynamics. These 
are equivalent to replica symmetric (RS) solutions given by using the replica method isl ]. 
Recently in the Hopfield model, BoUe et al compared the transient dynamics of the path- 
integral method with those of the signal-to-noise analysis only for a few time steps in the 
dynamics They have pointed out that the signal-to- noise analysis is exact up to time 
step 3 and inexact to step 4 or beyond. 

In order to discuss the relation between the path-integral method and the signal-to-noise 
analysis in more detail, we analyze a finite-step sequence processing model, which includes 
the Hopfield model and the infinite-step sequence processing model in special cases. In the 
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finite-step sequence processing model, the steady states of tlie system become limit cycles. 
Since the finite-step sequence processing model can store limit cycles in the dynamics, the 
properties of the system are periodic and dynamic essentially like the infinite-step sequence 
processing model. Moreover, the statistical properties of the finite-step sequence processing 
model are more complex than the infinite-step one. Since the period of the limit cycle 
is finite, the network takes the same states repeatedly. Namely, the finite-step sequence 
processing model has the theoretical difficulties of both the Hopfield model and the infinite- 
step sequence processing model. In this point of view, it would be very interesting to 
theoretically discuss the properties of the finite-step sequence processing model. 

In this paper, we have exactly derived the transient dynamics of macroscopic recursive 
equations with respect to the finite-step sequence processing model by means of the path- 
integral analysis. Until now, only in the infinite-step sequence processing model, which has 
no self-interactions, Diiring et al derived the stationary state equations of the order param- 
eters by using the path-integral analysis 3]- The transient dynamics of various disordered 
systems can be also analyzed by using the path-integral method. Therefore, it is important 
to derive stationary state equations of the order parameters from the macroscopic recur- 
sive equations. Assuming the Maxwell construction ansatz, we have succeeded in deriving 
the stationary state equations from the macroscopic recursive equations with respect to the 
model, which has self- interactions, i.e., the finite-step sequence processing model. 

We also analyzed the finite-step sequence processing model by means of the signal-to- 
noise analysis (SCSNA). The stationary state equations given by the path-integral analysis 
are equivalent to those of the signal-to-noise analysis. This result corresponds to the fact 
that the replica method and the signal-to-noise analysis give completely equivalent results 
in the stationary state analysis of the Hopfield model. Namely, the transient dynamics given 
by the signal-to-noise analysis gives an exact solution in both the stationary state and the 
first few time steps in the dynamics. 

II. DEFINITIONS 

Let us consider a system storing an infinite number of limit cycles with / finite steps. The 
system consists of Ising-type spins (or neurons) (Tj = ±1. We consider the case where 
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N ^ oo. The spins updates the state synchronously with the probabihty: 

1 



Proh[ai{t + l) = -ai{t)] 



1 — (Tj(t) tanh/3/ij(t) 



N 



(1) 

(2) 



hi{t) =^Jijaj{t) + ei{t), 

where /3 is the inverse temperature, P — 1/T. When the temperature is T = 0, the updating 
rule of the state is deterministic. The term 9i{t) is a time-dependent external field which 
is introduced in order to define a response function. The interaction stores p random 
patterns ^'^''^ = (^i''^, • • • , Cify so as to retrieve the patterns as: 



(3) 



sequentially for any /xth limit cycle. For instance, the entries of the interaction matrix 
J — {Jij) are given by 



J, 



1 



p/i I 



T^EE^r^^r^ (4) 

U=l fl=l 

where the pattern index n are understood to be taken modulo /. Since the number of limit 
cycles is p/l, the total number of stored patterns is p. The number of stored patterns p 
is given by p = aN, where a is called the loading rate. In our analysis, the number of 
steps for each limit cycle I is kept finite. Each component of the patterns is assumed to 
be an independent random variable that takes a value of either +1 or —1 according to the 
probability: 



Prob[e 



±11 = 



1 rj. 



For the subsequent analysis, the matrix J is represented as 

J 

where the p x N matrix ^ is defined as 

I = . . . . . . . . . 

and the p x p matrix S is defined as 



p/l,l\T 



(5) 



(6) 



(7) 



S 



(s' 



\ 







(8) 



S') 



and the I x I matrix is defined as S' — {S'^j,) — ((^/i,(i/+i)modi)- When I — 1, i.e., 5 = 1 (1 is 
the unity matrix), the system is equivalent to the Hopfield model. 



III. PATH-INTEGRAL ANALYSIS 



During et al discussed the sequential associative memory model by means of the path- 



integral analysis [1^. In this section, we introduce macroscopic state equations for the model 
with a finite temperature T > 0, according to their paper. 

In order to analyze the transient dynamics, the generating function Z[il)] is defined as 

cr(o),-,cr(t) 

where t/) = {ip{0), ■ ■ ■ , ipit — 1))^ and the state cr(s) = (cri(s), • • • , (T7v(s))^ denotes the state 
of the spins at time s. The probability p[cr(0), ■ ■ ■ , (T[t)] denotes the probability of taking 
the path from initial state cr(0) to state a(t) at time t through cr(l), cr(2), ■ ■ ■ , cr(i(: — 1). 
As Q shows, the generating functional entails the summation of all 2*^*+^^^ paths which 
the system can take from time to t. One can obtain all the relevant order parameters, 
i.e., the overlap m(s), the correlation function C{s,s') and the response function G{s,s'), 
by calculating the appropriate derivatives of the above functional and letting i/j tend to 
afterwards as follows: 

C{s, s') = - hm 1 V , ^'^!:^\ (11) 

Using the assumption of self-averaging, we replace the generating functional Z['ili\ with its 
disorder- averaged generating functional Evaluating the averaged generating function 

^[1/?] through the saddle point method, we obtain the following saddle-point equations for 
the order parameters of ()1()|1 - |T^ in the thermodynamical limit, i.e., N 00 (See Appendix 

El). 

m{s) = < ^ct(s) >, (13) 

C{s,s') = <(T(s)a(s') >, (14) 
_ d « a{s) » 

09(8') ■ ^^^^ 

The average over the effective path measure is given by 

< g{cT, «)>=(/ Vv Tr g{cT, v)p[a{0)] Y[hl + a{s) tanh/5/i(o-, v, s - (16) 



Vv = 17 

^(27r)*|i?| 



Tr = E ^ (18) 

a(o),-,<7{t)e{-i,i} 



h{cT, V, s) = f +^m(s) + e{s) + ^/^v{s) + (rcr)(s), (19) 

l-l 

R = J2 J2 G"^'+'^C(G"f)"'+^ (20) 

a=0 m,n>0 
l-l 

T = jY, e^'''^l\\ - e^^'^'I^G]-\ (21) 

withr = K ,Q = -^aiR and p[(t{0)] = |[l+a(0)m(0)] which is the initial spin probabihty. 
The operator < ■ >^ denotes the average over the condensed patterns. The term (rcr)(s) 
denotes the sth element of the vector Tcr. The vectors cr and v denote cr = {a"(0), ■ ■ ■ , cr()f:)} 
and V = {f (0), ■ ■ ■ , v{t—l)}, respectively. Equations (fT^ - (PT|) entirely describe the dynamics 
of the system. The term ns=i |[1 + ^'"(s) tanh/5/i(cr, v, s — 1)] in cannot be factorized 
with respect to spin variables at different times. Calculation of the spin summation of ()lfj|l 
requires an exponential time 0(e*) at time t. In the infinite-step sequence processing model, 
local field h{(T,v,s) depends on only spin variables at time s [IJ]. Therefore the term 
n*=i i[l + (^{s) tanh/5/i(<T, v,s- 1)] can be factorized, so the spin summations can be taken 
easily. 



IV. THE STATIONARY STATE 



In this section, we inspect time-translation invariant solutions of our macroscopic equa- 
tions (fT!?|) - (fTH|) for the deterministic dynamics, i.e., f3 —>■ oo (T = 0). The time-translation 
invariant solutions will describe motion on a macroscopic limit cycle: 

m{s) = m, 

< C{s,s')=C{s-s'), (22) 
^ G{s,s') =G(s-s'), 

with d{s) = 9. Now, we disregard the transient states. Note that the condition of ()22|) 
includes an unspoken condition that the transient states are disregarded. Therefore, we put 
that the dynamics is already in the stationary state at time s = under this assumption. In 
the zero noise limit, i.e., T = (/3 — oo), the dynamics becomes deterministic. Therefore, 
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we also assume that the system takes a fixed path as 

a{s + l) = a{s), (23) 

for any time s > 0. The path which the system takes after time s > can be described as 

a{s) = r/„ (24) 

by only / constants rjo, ■ ■ ■ ,rii^i G { — 1,1}. The pattern index s of the constants ris is 
understood to be taken modulo /. Note that it is not necessary to calculate these constants 
{rjs} explicitly. When the variable transformation 

x{s)=r]Ms), (25) 

is carried out to spin variables o"(s), the transformed spin variables x{s) take same value for 
any time s, i.e., x{s) = xi^') for any s,s'. Equation (fTBj) means the expectation of g{(T,v) 
with respect to the path probability. In the zero noise limit, equation (fT^ becomes 

< g{c7, v)'>=n Vv Tr g{(T, v)p[a{0)] J] S^{s),sgnh((T,v,s-i)) , (26) 

When the spin variables have periodicity as (t(s + /) = cr(s), the Gaussian random fields are 
also deterministic as v{s + l) = v{s) (See Appendix IB]) . For any function 0({cr(s)}) and any 
constants Cq, ■ ■ ■ , Cf G { — 1,1}, the following identity holds: 

Y: 0(a(O),---,a(t))= 0(coa(O),---,Qa(t)). (27) 

cr(0),-,o-(t)e{-l,l} a{0),-,a{t)e{-l,l} 

Applying ^I7\i to and substituting (j2SI), we obtain 
<t:g{(T,v) > 

t 

Vv Tr g{{coa{0), ■ ■ ■ , Qa(t)}, v)p[coa{0)] [] 4.<.(s),s 



s=l 



Vv Tr g{{coVoX{0), ■ ■ ■ , Ctr]tx{t)}, v) 



t 



with 



xp[cor/oX(0)] n ^c.r, sX{'^),sgnh{{coVox{0),--,ctVtxit)},'V,s~l) / ; 

s=l ' 4 

Tr g{{xiO),---,xit)},v)p[xiO)]l[5^^s),sgah{{x{o),-,x{t)},v,s-i))^, (28) 



Tr = Y = E • (29) 

riox{o),-,vtx{tM-hi} x(o),-,xWe{-i,i} 



In derivation of ()28|) . we put the constants {cs} as Cg = rjs for all s. Generality is kept even 
if Cs = rjs for all s. Namely, with respect to the transformed spin variable xi^), the effective 
single spin described by is 

X{s)= sgnh{{xiO),---,x{t)},v,s-l). (30) 

The transformed spin variables x{^) cire deterministic even if (jHn|l includes the Gaussian 
random fields v, since the Gaussian random fields are deterministic. In order to get rid 
of the self-interaction, we assume the Maxwell construction ansatz. Using the identity 
x{s) = x{^') foi' any s, s' to (IHUj) and applying the Maxwell construction, we get 

X{s) = sgn [Cm{s - 1) + e{s - 1) + V^v{s - 1) + E r(s, s')x{s)] 

s'<s 

= sgn [em{s - 1) + 9{s - 1) + V^vis - 1) + x{s) E r(s, s')] 

s'<s 

= sgnh{v,s -1), (31) 
with h{v, s) = ^'+^m{s) + e{s) + y/av{s). Substituting (|SH) into we obtain 

< g{(T,v) :^=/[vv Tr g {x, v)p[xm ]]_ ^X, Hv, s-i)) ■ (32) 

Thus, we can get rid of the self-interaction in the single spin problem by using the Maxwell 
construction in the zero noise limit, i.e., T = 0. Since (j32p can be factorized with respect 
to the tranformed spin variables x{^) different times, we can easily perform the spin 
summations. After simple rescalings we arrive at 

m{s) = (^C J sgn h{v,s-l)^^, (33) 

C{s,s') = 6s,s' + [l-6s,s']Qvv[sgnh{v,s-l)][sgnh{v,s' -1)]^^, (34) 

G{s,s') = hm /jjl - (^J Vvtanh^ (3h{v,s (35) 

We now calculate the matrix R under the condition of (12211. Since the matrices G and C 



become Toeplitz matrices (especially C is symmetric) under this conditions, C and G can 
be approximately regarded as commuting matrices, i.e., CG = GC. Therefore, the matrix 
R simplifies to 



R = Y^G^i E G"^'(Gt)"'c)(G^)'^ 

a=0 m,?i>0 



E G^[[l - G']-^[l - {G^y]-^C){G^y 

[1 - GG^]-'[1 - {GG^y][l - G']-^[l - {G^yr^C, (36) 
9 



We consider the persistent parts of C(r) and R{t) as C(r) — g and R{t) r for r — oo 
and also consider the non-persistent parts C(r) and R{t) 0, i.e., C(r) = g + C(r) 
and -R(r) = r + -R(t). Upon rewriting G{t) = /?5t-,i[1 — q] and r = qp given by (jSS)), we 
obtained 



U Dz -Dvsgnh{v,0\z] 



Dz / X'D[sgn/i(D, r|2;)][sgn/i(D, 0|2;)] 



(37) 
(38) 



q = Ymvl Dz Vv iaxih.^ (3h{v,Q\z) 

P^oo \J J / ^' 



(39) 

r = qp, (40) 

from dSl-dSSI) with Vv = dve'^^'"-^ ^[(27r)*|^|]-i/^ Dz = ^^e'^'/^ g^^d /i(v,r|2) = 
^"^m + 6^ + Zy/aqp + y/av^r). The matrix G is given by 

^ ■■■ o^ 

G(l) ■■■0 
G= G(l) •••0 . (41) 



\ ■■■0 G(i) y 

from p5|) . Equation p6|) can be changed to 

[1 - (G^)' -G'- {G^Gy][l - {GG^y]-'[l - GG^]R = C. 



(42) 



The identity GG^ ~ ^(l)^! holds in the case where time s is sufficiently large, so the 
left-hand side of becomes 

[1 - {G^y - G' - (G^G)'][1 - {GG^y]-^[l - GG^]R 
( D{0) D{1) ■■■ D{t-1)\ 
D{1) D{0) ■■■ D{t-2) 



(43) 



where 



yD{t- 1) D{t-2) ■■■ D{0) j 



DiO) = il + gf)g2RiO)-2g,g2Ril), 

D{s) = (1 + gj)g2R{s) - g^Ris -1) + R{s + 1)], (0 < s < t - 1) 
D{t-1) = {l + gj)g2R{t-l)-2g,g2R{t-2), 



(44) 
(45) 
(46) 
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with gi = G{iy and (72 = [1 — G'(l)^]/[1 — (7(1)^']. Since D and C are symmetric Toeplitz 
matrices, they can be diagonahzed by using the discrete Fourier transformation (See Ap- 
pendix [nj. The Fourier transformations (or the Lattice Green's functions) Dk,Ck of the 
matrices D, C are given by 

t-i 



D, J2hl+gl)g^R(r)-g,g,[R{T-l)-RiT+l)]y''\ (47) 

r=0 ^ 

Ck = EC(r)e*'=^ (48) 

T = 

For any wave number k, Dk = Ck holds when D = C. Taking the hmit s — > 00 about 
-Do = Cq, the following relationship is obtained: 

^ l-G(l)^^ 

By working out the remaining integrals over v and setting 6 = 0, we finally obtain the 
stationary state equations of the order parameters as follows: 



U = J^e'^p, (51) 
V nap 

^ ~ (l-f/2)(l-f/')2' ^ 

with g = 1, g = 1 and U = G{1) where erf(-) denotes Error function defined by erf(a;) = 
- fn e~'^^du. It turns out that these stationary state equations (ISUll - dS^ given by this exact 
loLon a. e,.va.e,. to tKose of tKe s,„a.-to-.o,se a.a>..s (slA^pend^E ji K,.e 
H shows that the storage capacity ac and the number of patterns per one limit cycle s. 
Figures El and El compare the theoretical results and computer simulations for / = 3, 7 (The 
number of spins is = 3000, the number of iterations is 11). The data points and error bars 
show the results of the computer simulation. With respect to the computer simulation in 
figure 121 and El the stationary overlaps are defined as m(lOOZ) and m(50Z), respectively. It is 
confirmed that the theoretical results are in good agreement with the computer simulations. 
Storage capacity monotonously increases from ac = 0.138 (/ = 1) with the number of steps / 
. In the large I limit, storage capacity finally converges to ac = 0.269, which coincides with 
the theoretical result of for the infinite-step sequence processing model given by Diiring et 



al 



12j . The original properties of the finite-step sequence processing model appear as long 
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as the number of steps of a limit cycle I has order I — 0{1). In the case that I has the order 
more than the properties are the same as the properties of the infinite-step sequence 

processing model. 



0.28 - 




0.12 



0.1 ' — ' ' ' ' ' ' ' 

1 2 4 6 8 10 12 14 

loading rate 



FIG. 1: the storage capacity Oc and the number of patterns per one limit cycle s. 
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' ' ' ' ' — ^ ' ' ' ' ' 

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 



loading rate 



FIG. 2: Computer simulations {I = 3,N = 3000, 11 times): the overlap m and loading rate a. 



V. CONCLUSIONS 



We exactly analyzed an associative memory model storing an infinite number of limit 
cycles with finite steps by means of the path-integral method. In the case where the dynamics 
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simulation 



0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 
loading rate 

FIG. 3: Computer simulations {I = 7, N = 3000, 11 times): the overlap m and loading rate a. 

is deterministic, the statistical properties are simplified like those of the infinite-step sequence 
processing model by using Maxwell construction. We also derived the macroscopic equations 
for stationary states at T = 0. 

We obtained the dependence of storage capacity (etc) on the number of patterns per one 
hmit cycle (l). At Z = 1, storage capacity is etc = 0.138, as in the Hopfield model. A 
storage capacity monotonously increases with the number of limit cycles, and converges to 
ac = 0.269 at / ~ 10. The original properties of the finite-step sequence processing model 
appear as long as the number of steps of the limit cycle has order / = 0(1). 

In the case where the dynamics is deterministic, we also derived the stationary state 
equations by using Maxwell construction. The stationary state equations are equivalent to 
those of the signal-to-noise analysis. This means that the signal-to-noise analysis apphed 
to the stationary state is exact in spite of including errors in the middle of the transient 
dynamics in the zero noise limit. 
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APPENDIX A: DERIVATION OF ORDER PARAMETER EQUATIONS 



Diiring et al discussed the sequential associative memory model by using the path-integral 



method [1^. Here , we discuss for the model with finite temperature T, according to their 
paper. Most of the technical detail to derive order parameter equations is almost identical 
to the paper of Diiring et al [3] • 

The generating functional contains both condensed and non-condensed patterns. 

We isolate the non-condensed ones by introducing the local field h and the variables x, y: 

^ ^ J dhdh ^^ihM\h^{s)-y^J,,a,{s)-9M] ^^^^ 

J (27r)(P-i)* ' ^ ' 

j (27r)(P-i)* ■ ^ ' 

We isolate the various relevant macroscopic observables by inserting integrals over appro- 
priate 5-functions: 

i!!:^e^E.<.-WhW-^E,«.^'^-«W], (A4) 



(27r/A^) 

1 = ( J^^e'Ts<M4Hs)-j,T.il-'^'~Us)]^ (A5) 



(27r/A^) 

~ ' i27r/ NY' ' ^ 

dQdQ ,y-^ Q(s,s')[Q{s,s')-^Y.^ I A 7^ 

(27r/iV)*^ ■ ' ^ ^ 

~ J {271 /NY' • ^ ^ 
The generating functional which for N ^ oo will be dominated by saddle points. We obtain 

Z[iP]= J drndmdkdkdqdqdQdQdKdke^^'^+'^+^^+^^^'^'\ (A9) 

by substituting (IMl)- (IX8|l into Q, where 



\l/ = rh{s)'m{s) + k{s)k{s) — m{s)k{s) 
s<t '- 

+i q{s,s')q{s,s') + Q{s,s')Q{s,s') + K{s,s')K{s,s'] 

s,s'<t '- 

* = ^ElnTrp.KO)) J {dhdh} 



(AlO) 
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J2s<t [/9o-(s+l)-ln 2 cosh ph{s 



xe 



(All) 



dudv 



N J (27r)(P-*)* 

Xe~5 I]^>1,^>1 Z]s,s'<J"'-'^M(s)'9{«.s')''>^,M{*')+">',M(s)'?^(S:«')f--,M(s')] 
I I]^>1,M>1 I]s,s'<t^''.A'('')-^(*'''')"^,M{«')+'">^.M(«)'?{«.s')f>',M(s')] 



xe 



(A12) 



dhi(s)dhi{s) 
2^ ■ 



with the shorthand {dhdh} = Hi 

In the hmit N oo, the integral ()A9j) will be dominated by saddle point of the extensive 
exponent \E' + $ + fi. The saddle-point equations which are derived by differentiation with 
respect to integration variables {m, rh, k, k, q, q, Q, Q, K, K} are as follows: 

k{s) = 0, 
q{s,s') = 0, 



m[s) 
Q{s,s') 
m(s) 



kis) = lim - 5: < ais)^-' > 



q{s,s') 
K{s,s') 
Q{s,s') 
K{s,s') 



C{s,s') = lim < ^i^M^') >i 

t 

iGis s') = ^\im -y ^ ^ ^^'^ 



(A13) 
(A14) 
(A15) 



i lim 



on 



g_0 dK{s,s') 

on 



saddle 



dK{s,s') 



saddle 



(A16) 

(A17) 

(A18) 

(A19) 
(A20) 



where /Isaddie denotes an evaluation of a function / at the dominating saddle-point, < ■ >i 
denotes 

Tia J{dhdh}Wi{(T, h, h)f{(T, h, h) ' 



< f{cT,h,h) >, 



Tia- J{dhdh}Wi{(T,h,h) 



(A21) 



Es<t {Ms+lMs)-\n 2 cosh Ph{s)) 



(A22) 



and < ■ >^ denotes the average over the condensed patterns. We now calculate the right- 
hand sides of ()A18|) and ()A19|) . The eigenvalues of a matrix S are given by = e^'^^^/^ (the 
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multiplicity is p/l) from |A1 — = |A1 — S"|^/'. Since the matrix S' is a unitary matrix, 
i.e., S"^S" = 1, the matrix S is also unitary, S'^ S = (iia.g{S'\- ■ ■ , S''^)(iicig{S' ■ ■ , S') = 
dmg{S'^S',---,S'^S') = 1. A (/i, /i)-element of (5+)™5" becomes = 5^„, by 

using the unitarity of S, where 5mn denotes Kronecker's delta function. The identity (S)' = 
{S^Y = 5 is estabhshed because {Sj = {S'^Y = S' . Therefore, the identity [(5t)"^5"]^^ = 
6mn holds for the following value (m, n) : 

m = m'l + a, 

(A23) 

n = n'l + a, 

with m', n' G {0, 1, ■ • •} and a G {0, 1}. The multiplicity of the eigenvalues of the 

matrix S' is 1, so rank(S" — s^l) = 1 — 1. Hence, rank(S' — s^l) = (p//)rank(S" — s^l) = 
p — p/l. This means that the matrix S can be diagonalized by an appropriate non-singular 
matrix as 

p/l p/l 
diag(so,---,So, 

By working out the saddle-point equation ()A18|1 . Q becomes as follows: 

Q{s,s') = ~m J2 Jim 5]{(5®G)"^[l®C](5t®Gtr}^^(s',s). (A24) 

m,n>0 M^P 



Hence Q is given by 

1 

^ a=Om',n'>0 

We define a matrix F = S Cg) as having matrix elements T^^/{s,s') = S^^'R{s, s') for 
G and s,s' G {0,---,t — 1} where y = Tx will operate as = 

I]/^'>t I]s'<t 'S'^t/i'-R('5, s')a;^/(s') for each {fi,s). Equation ()A19|1 reduces to 

K{s,s') = -la—^—— lim i{lndet[l®l-5t®Gt]+lndet[l®l-5®G]} 
2 dG{s, s') p^°° p 

= n - In n det[l - e^^^^/^Gt] (A26) 



k = j Y. e'"'^/'[l - e'"'^/'Gt]-\ (A27) 



Ai=0 

i-1 



p=0 



Replacing ^l'^ — * we obtain the order parameters of (fT3j) - (fT3j) . 
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APPENDIX B: THE PERIODICITY OF THE GAUSSIAN RANDOM FIELD 



Using time-translation invariant ansatz, i.e., C{s, s') = C {s — s') , C{s) also have period- 
icity as 

C(s + l) = Cis), (Bl) 

for any s, when spin variables a{s) have periodicity, i.e., a{s + I) = cr{s). It is confirmed 
that the vector v obeys the Gassian distribution with mean and covariance matrix R 
from equation The covariance matrix R of the Gaussian random fields v is given by 

(jnSl)- Since the matrix R is also symmetric Toeplitz matrix, we can put R{s, s') = R{s — s') 
where R{s, s') are the elements of the matrix R. When C{s + I) = C{s), we approximately 
get R{s + I) = R{s). The correlation coefficient between cross-talk noise v{s) and v{s + I) 
becomes 

Corr(.(. + /),.(.)) = CovK^ + ^ + ^) = i, 

^Vivis + l))V{v{s)) ^R{s + l,s + l)R{s, s) 

for any s, s'. The v{s) distribution and the v{s + /) distribution have the same mean and 
variance, i.e., E{v{s)) = E{v{s + /)) = and V{v{s)) = R{s, s) = R{0) = R{s + l,s + 1) = 
V{v{s + I)). Therefore, the identity 

v{s + l)=v{s), (B2) 

holds for any s. Hence, the Gaussian random fields v{s) also have periodicity as v{s + 1) = 
v{s) when spin variables have periodicity as a{s+l) = cr(s). Therefore, v{s) can be considered 
as quenched noise. After the states f (0), ■ ■ ■ ,v{l — 1) occur at random, v{s) continues taking 
the same values periodically. Therefore, we can assume that v{s) is deterministic for a fixed 
site. The Gaussian random fields v{s) are only distributed with respect to site index i. 

APPENDIX C: THE FOURIER TRANSFORMATION OF THE SYMMETRIC 
TOEPLITZ MATRIX 

Symmetric Toeplitz matrices can be diagonalized by using the discrete Fourier transfor- 
mation. The Fourier transformation was used to obtain the identity of The Fourier 
transformation is defined as 

4 = ^Ee.e-^^ (ci) 
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and the inverse Fourier transformation defined as 



(C2) 



where k denotes wave number and its degree of freedom is t. Each component of k takes the 
value 0, |7r, |7r, • • • , ^^*~^^ 7r. The following symmetric Toeplitz matrix can be diagonalized 
by using the Fourier representation: 



/ 



D 



Do ■■■ Dt.i 

Di Do ■■■ Dt-2 



\ 



y Dt-l Dt-2 ■ ■ ■ Do J 

The Fourier representation of the quadratic form ^ becomes 

i=\ 3=1 

T=Oj = l 

= E E (4? E ^>^^^'''')M^, ^ ik^e^'^^'-'^ 



(C3) 



(C4) 



where the index j of the variables is understood to be taken modulo t. Therefore, the 
Fourier transformation of the symmetric Toeplitz matrix D = (-Dr) is given by 



(C5) 



This transformation is also called the lattice Green's function. 



APPENDIX D: THE SIGNAL-TO-NOISE ANALYSIS OF THE FINITE-STEP 
SEQUENCE PROCESSING MODEL 

We discuss the stability of limit cycles by means of the signal-to-noise analysis. Let us 
consider the following deterministic synchronous dynamics: 

N 

^r' = i^(E>^^.4)' (Di) 
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where the x* represents the state of the i-th neuron at time t and F(-) denotes an output 
function. The retrieval state converges to some hmit cycle with / steps. We introduced the 
Poincare map to get the states every / steps in the steady state. The periodic state can be 
transformed into a stable state by using this map. Hence, we can discuss the properties of 
a stability of limit cycles. 

Let us consider the case of convergence to periodic states of the limit cycle retrieval. We 
assume x* = xl~^ in ()D1|) . The overlap between the /i-th memory pattern ^'"'^ of the u-th 
limit cycle and the network state x is defined as 



1=1 



The dynamics (jPlll can be rewritten as 

= F{h^ (D3) 

N p/l 

j = l U=l IJL 

where h\ is a local field. Now let us consider the case to retrieve the 1st limit cycle 
/i G {1, ■■■,/}. We assume the memorized pattern of another limit cycles ^'^^j ^ 1 
does not have a finite overlap. We consider retrieval solutions in which ~ C'(l), and 
m^^ ~ 0(l/v^), > 2. We assume that the components Xi of the equilibrium state x are 
independent on the unit number i in the limit N oo. It is necessary to assume that the 
self- averaging property to holds so that the site average can be replaced by an average over 
the random patterns and random variable x. In this situation the overlap m^^ need not to 
be a random variable. 

<M = K^. + Utmi-\ (D5) 



m 



i 



Y.a''xr'\ (D6) 



Ut = (D7) 



where 



x--=F{ Y: (D8) 
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By applying ()D5|) repeatedly, we obtain 



m 



,t-i 



(DIO) 



Since the retrieval state is assumed to be steady, i.e., x 
holds. Therefore the overlap becomes 



t-i 

i 



xl, the identity m* ' 



Ufl — l 



m 



l-l 



m 



t-k) 



v-1 



fc=0 



VfJL — k 



k=l k'=0 



(Dll) 



Substituting ()D6|1 into ()D11|1 . we obtain 



i-i 



m 



TV 



N l-l k-1 N 

j=l k=l k'=0 j=l 



X, 



(D12) 



Replacing -> and 
be rewritten as 



m^, and substituting ()D12|) into ()D4|) . the local field /;.* can 



EC 



+^(1 - n f^*-^)-^ 



aN/l I 
k=0 [i^>2 At=l 

aN/l I N 

E EE^r^'er^f"^ 

!/>2 At=l JT^j 
Z-1 k-1 aN/l I N 

+ Edl Ut-k') E j^j^er+^ef^'^^'^^""'^ 



fc=0 fc'=0 

l-l 



+^(i-nc.-* 

K = 



-1 



ufi+1 f^ufi—k t—k{v^—k) 



k=D k'=0 u>2 fi=l j^i 

The first term in ()D13jl is regarded as the signal. The second and the third terms are 
regarded as the mean and the variance of the crosstalk noise, respectively. In the second 
term, if the suffix isyU + 1 = /i — /c( mod/), ^i^^^ and ^-^e represented the same patterns. 
Since these terms consist of uncorrelated random variables with the order 0{1/ \/N), the 
other terms in the second term can be omitted. The term of = / — 1 is only one remaining 
of the terms with k G {0, ■■■,/ — 1}. Hence we can estimate the second term as 

1 1-2 l-l 



fc'=0 fc'=0 



aN/l I 

EE 

u>2 ti=l 



Ufl+l^UfJ.-1 + l t-l + l{l^fl-l + l) 
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(D14) 



k'=0 



k'=0 



We assume the third term is normally distributed with mean and variance because of 
the independence of x^^'^^^ and C.i'^- The variance af of the cross talk noise is estimated as 



at = a 



t-k' 



k'=0 



k=l k'=0 



2t~k 



where 



N 



^t-k\2 



(D15) 



(D16) 



The local field hi is obtained by setting Zi ~ A^(0, 1) in ()D13jl . 



1-2 



(D17) 



^l=l k'=0 k'=0 

The self-averaging property is assumed. Replacing Xi —>■ Y and — > C,'^, we obtain the 
macroscopic equations as follows, 



/oo 
-CXD 

/OO Q 
Dz » 
-oo 

1 



:i - n Ut-k' 



k'=0 



r = a(n f^t-fc')(l- UUt^k 

k'=0 k'=0 



k=l k'=0 
1 



(D18) 

(D19) 
(D20) 
(D21) 

(D22) 
(D23) 



Now let us consider the case that the state of memory retrieval is periodic. We can set 



ml = m6ut, qt = q, Ut = U, at = a, (t = l, ■■■,/), 



(D24) 



where 6^t denotes Kronecker's delta. Finally, we obtain the macroscopic equations as follows: 



Y{e^';z) 
m 

q 



F{^'^^m + TY{^'^^;z) + az) 

/oo 
Dz 
-oo 

/" Dz <^Y{e+';zf > 

J — oo 



(D25) 
(D26) 

(D27) 
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u 



1 roo 

- Dz z <^Y{^'+';z):$> (D28) 
cr j-oo 



' — oo 

t21 



r = ^3^. (D30) 



Setting F(-) = sgn (■) and using Maxwell rule we obtain (I^Ujl - lj^ as follows: 

U = J^e~^p, (D32) 
V nap 

1 - 

= (i_f/2)(i_t/;)2' (D33) 
where cr^ = apg, ^7 = 1 and sgn (■) denotes the sign function ( sgn (x) = 1 for x > 0, —1 for 
X < 0). Thus, we find that these stationary state equations of the order parameters given 
by the signal-to-noise analysis (SCSNA) are equivalent to those of the path-integral analysis 
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